On Centralized and Distributed Mirror Descent: Convergence Analysis Using Quadratic Constraints

نویسندگان

چکیده

Mirror descent (MD) is a powerful first-order optimization technique that subsumes several algorithms including gradient (GD). In this work, we leverage quadratic constraints and Lyapunov functions to analyze the stability characterize convergence rate of MD algorithm as well its distributed variant using semidefinite programming (SDP). For both algorithms, consider strongly convex nonstrongly assumptions. centralized problems, construct an SDP certifies exponential rates derive closed-form feasible solution recovers optimal GD special case. We complement our analysis by providing explicit $O(1/k)$ for problems. Next, numerically whose dimensions are independent network size. To best knowledge, numerical has not been previously reported in literature. further prove setting. Our experiments on problems indicate framework superior compared existing GD.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Convergence of Online Mirror Descent Algorithms

In this paper we consider online mirror descent (OMD) algorithms, a class of scalable online learning algorithms exploiting data geometric structures through mirror maps. Necessary and sufficient conditions are presented in terms of the step size sequence {ηt}t for the convergence of an OMD algorithm with respect to the expected Bregman distance induced by the mirror map. The condition is limt→...

متن کامل

Distributed Mirror Descent over Directed Graphs

In this paper, we propose Distributed Mirror Descent (DMD) algorithm for constrained convex optimization problems on a (strongly-)connected multi-agent network. We assume that each agent has a private objective function and a constraint set. The proposed DMD algorithm employs a locally designed Bregman distance function at each agent, and thus can be viewed as a generalization of the well-known...

متن کامل

Convergence Analysis of Distributed Stochastic Gradient Descent with Shuffling

When using stochastic gradient descent (SGD) to solve large-scale machine learning problems, a common practice of data processing is to shuffle the training data, partition the data across multiple threads/machines if needed, and then perform several epochs of training on the re-shuffled (either locally or globally) data. The above procedure makes the instances used to compute the gradients no ...

متن کامل

Data-Distributed Weighted Majority and Online Mirror Descent

In this paper, we focus on the question of the extent to which online learning can benefit from distributed computing. We focus on the setting in whichN agents online-learn cooperatively, where each agent only has access to its own data. We propose a generic datadistributed online learning meta-algorithm. We then introduce the Distributed Weighted Majority and Distributed Online Mirror Descent ...

متن کامل

Global Convergence of Steepest Descent for Quadratic Functions

This paper analyzes the effect of momentum on steepest descent training for quadratic performance functions. Some global convergence conditions of the steepest descent algorithm are obtained by directly analyzing the exact momentum equations for quadratic cost functions. Those conditions can be directly derived from the parameters (different from eigenvalues that are used in the existed ones.) ...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: IEEE Transactions on Automatic Control

سال: 2023

ISSN: ['0018-9286', '1558-2523', '2334-3303']

DOI: https://doi.org/10.1109/tac.2022.3230767